Learning Concepts by Synthesizing Minimal Threshold Gate Networks

In feed-forward layered neural networks, intermediate concepts are identified with hidden units. Each hidden unit represents an intermediate concept, and its function is to be derived by some learning algorithm. In general, the smaller the network, the better the generalization performed, as long as it is large enough to learn the required mapping [Huyser & Horowitz, 1988]. Empirical evidence suggests that, in many cases, learning with the absolute minimum number of units is far more difficult than if some extra hidden units are allowed [Rumelhart & McClelland, Eds. 1986].